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Background 



In the pragmatic-explanatory continuum, a randomized 
controlled trial (RCT) can at one extreme investigate 
whether a treatment could work in ideal circumstances 
(explanatory), or at the other extreme, whether it would 
work in everyday practice (pragmatic). How explanatory 
or pragmatic a study is can have implications for clini- 
cians, policy makers, patients, researchers, funding bod- 
ies, and the public. There is an increasing need for stud- 
ies to be open and pragmatic; however, explanatory 
trials are also needed. The previously developed 
Pragmatic-explanatory continuum indicator summary 
(PRECIS) was adapted into the Pragmascope tool to assist 
mental health researchers in designing RCTs, taking the 
pragmatic-explanatory continuum into account. Ten 
mental health trial protocols were randomly chosen and 
scored using the tool by three independent raters. Their 
results were compared for consistency and the tool was 
found to be reliable and practical. This preliminary work 
suggests that evaluating different domains of an RCT at 
the protocol level is useful, and suggests that using the 
Pragmascope tool presented here might be a practical 
way of doing this. 
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.he randomized controlled trial (RCT) has long 
been recognized as the most robust technique for eval- 
uating the effects of mental health care interventions. 1 
Sometimes these trials are impossible, sometimes uneth- 
ical, and sometimes impractical. The first RCT was the 
1948 Medical Research Council Streptomycin Trial. 2 In 
the austere times of bankrupt, post-war England, just as 
the National Health Service was being established, it was 
suggested that a new drug would be of value for treat- 
ment of tuberculosis. The only equitable way to distrib- 
ute this scarce resource was through randomization and 
then, by the establishment of good evidence, to encour- 
age those funding health care to support its use. There 
are many interesting and important examples preceding 
this date, 3 but this landmark and courageous trial radi- 
cally changed the pathway of evaluation of medical 
treatments. 

Mental health has a fine tradition of using trials to eval- 
uate treatments. 4 The MRC Streptomycin trial coincided 
with the discovery of psychoactive compounds that were 
potentially therapeutic, as well as an increasing push 
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towards deinstitutionalization. Mental health profes- 
sionals discovered, discussed, and, largely, embraced the 
use of RCTs. Up to the advent of antipsychotic drugs 
such as chlorpromazine, psychiatric care had been 
likened to "little more than zoo keeping" 5 and, perhaps 
because of that stinging criticism, those undertaking the 
trials did not necessarily follow the path of that first 
tuberculosis trial. Factors combined to largely direct 
mental health trials along another route. There was the 
yearning for rigorous science, collective subspecialty 
insecurity, and also the needs of regulatory authorities. 
Mental health trials drifted towards use of as rigorous 
diagnoses as possible, rather rigid regimens of care and 
use of fine-grained outcome measures that are not usu- 
ally part of routine practice. This planted the RCT firmly 
in the realm of researchers, and there it has stayed. The 
needs of regulatory authorities did have to be met, but 
there was less consideration of needs of clinicians and of 
recipients of care and their families. 
This was not at all unique to mental health, but it took 
leaders in the fields of cancer care, 6 heart disease, 7 and 
perinatal medicine 8 to recall and refine the techniques 
of generous inclusion, simple treatment, and routine data 
collection that underpinned the MRC trial of 1948. 
Many examples now exist in these areas of RCTs where 
entry criteria are broad and encompass as many relevant 
people as possible, the treatment packages are those that 
would be given in everyday care, and outcomes are 
essentially routinely recorded data. Examples of such 
open work were rare in mental health until relatively 
recently. The description of "pragmatic" or "practical" is 
increasingly employed of trials in psychiatry or psychol- 
ogy but there are clearly different interpretations of 
what this really means. 

A recent series of papers has highlighted the problems 
in interpretation of the explanatory/pragmatic domains 
in trials and presented some practical solutions. 9 It is not 
a simple continuum from explanatory through to prag- 
matic. There are many elements of design that should be 
considered to allow a judgment to take place about 
whether a randomized trial is investigating whether, in 
ideal circumstances, a treatment could work (explana- 
tory) or, at the other extreme, whether this accessible 
treatment would work in everyday practice (pragmatic). 
This is not a purely academic exercise. There are good 
reasons to make these judgments. To use one example, 
funders, on receiving a proposal, may wish to consider 
whether the proposed trial fits with the ethos in which 



that support was proffered. For example one funding 
body may be interested in discovering potentially new 
treatments. In this instance, explanatory studies, under- 
taken in very rigorous circumstances with fine measures 
of outcome to highlight any — even modest — effects, 
may be best. On the other hand, another funder, using 
public money, may wish to consider whether the study 
is likely to produce evidence of practical importance 
regarding accessible treatments relevant to the major- 
ity of people suffering with the condition in that com- 
munity. 10 Here a much more pragmatic study would be 
desired. How explanatory or pragmatic a study (or a 
group of studies) is has also obvious and direct impli- 
cations for clinicians, policymakers, patients, and the 
public. 

The main goal of this study is to adapt the instrument 
described by Thorpe et al 9 (PRECIS) to assist 
researchers in making those judgments in the protocol 
stage of RCTs in mental health (the Pragmascope 
tool). 

Methods 

The Pragmascope tool 

This tool is based on the ten domains described in the 
development of the Pragmatic-explanatory continuum 
indicator summary (PRECIS). 9 It can be used to assess 
applicability of results from any given RCT, based on 
what was planned at the protocol stage. 
Each included RCT protocol 1119 was scored in ten 
domains by three independent reviewers (GT, KSW, 
CEA). The reviewers made a judgment and rated the 
protocol from 1 (most explanatory) to 5 (most prag- 
matic) by reading the details of the protocol. If the pro- 
tocol did not contain any information on which to base 
the decision, these domains were rated as zero. The aver- 
age scores for each included protocol were placed on the 
wheel diagram and the dots joined for visual clarity 
(Figure 1). 

Selection of RCT protocols 

We searched the Cochrane Schizophrenia Group Trials 
Register and Medline (November 2010) for references of 
RCT protocols and chose a random sample of 10 proto- 
cols dealing with schizophrenia, depression, post-trau- 
matic stress disorders, and psychiatric rehabilitation. 1119 
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Scoring the Pragmascope tool 

Three independent reviewers (GT, KSW, CEA) scored 
each included RCT protocol. The overall score can be 
from 0 to 50 and a diagram illustrating how open (prag- 
matic) or restrictive (explanatory) the study is likely to 
be was created using the average score of the three inde- 
pendent reviewers. 

Our initial interpretation of the scores was of 0 to 30 for 
an explanatory study investigating whether the experi- 
mental intervention will work in ideal circumstances and 
a total score >35 for a more pragmatic study focusing 
mostly on whether, in routine practice, an intervention 
has a meaningful effect. A total score between 31 and 39 
were interpreted as an interim where trial design bal- 
ances pragmatic and explanatory domains. 

Data analysis 

Mean and variance were calculated for each domain of 
the Pragmascope tool for each included RCT protocol 
using STATA (version 10). In addition, a weighted 
kappa for the domains was calculated using R. 



Results 

Table I presents the average score of the three raters in 
each one of the domains for each RCT protocols with a 
judgment based on the scores. Reliability among the 
three independent raters was high (weighted Kappa =0.72 
for categories 0-30, 31-39, 40-50) suggesting that this clus- 
ter of judgments might be useful to highlight and quantify 
important issues during the protocol stage of an RCT. 
We recognize that validity is a more problematic issue, as 
this does depend on the rater's perspective, but work is 
ongoing involving raters from very different backgrounds. 
In any case, we concur that consideration of these 
domains is useful 9 and suggest that the Pragmascope is 
one practical way of doing this. 

Discussion 

The world of RCTs has changed remarkably in the last 
10 years. Systematic reviewing of trials, now industrially 
undertaken through initiatives like the Cochrane 
Collaboration, 20 has highlighted issues with poor design 
and inconsistent reporting. These systematic reviews are 
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Figure 1. Examples of output. 

Reproduced from ref 9: Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD, Altman DG, et al. A pragmatic-explanatory continuum indicator 
summary (PRECIS): a tool to help trial designers. J Clin Epidemiol. 2009;62:464-475. Copyright ©2009 Elsevier 
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potent to guide care but are undermined by trial evi- 
dence that is difficult or impossible to apply in the real 
world. For mental health, studies of increasing pragma- 
tism are now being designed and undertaken. 2123 Such 
pragmatic, real- world, practical design can be dovetailed 
within explanatory studies or sit independently. With 
maintained systematic reviews guiding practice, 21 trans- 
parent priority setting for research funding for evalua- 
tive research, 3 and the push towards defining core out- 



come measures of agreed relevance in trials, 24 a great 
increase in pragmatic trial activity is likely. Of course 
explanatory trials have an important place in the port- 
folio of research, but the rigorously undertaken but 
highly pragmatic trial will give us the opportunity to 
learn much more about the real effects of the potent 
treatments we give. □ 

Acknowledgements: We would like to thank Dr K. Thorpe for consider- 
ation of our rating tool, and Dr B. Park for help calculating Kappa. 
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Table I. Average score of three raters for each one of the domains of the RCT protocol. 

Appendix. The Pragmascope (Figures 2 and 3) 

Explanation: This tool is based on ten domains described in the devel- 
opment of the Pragmatic-explanatory continuum indicator summary 
(PRECIS). 9 It can be used to assess applicability of results from any given 
randomized controlled trial. 

Instructions: For each of the ten domains please make a judgement and 
rate the protocol from 1 (most explanatory) to 5 (most pragmatic). Your 
score should be based on a thorough reading of the protocol. If the pro- 
tocol does not contain any information on which to base your decision 
we advise a default score of zero. If you feel your scores require justi- 
fication - please make comments in box provided. Mark score on rele- 
vant part of the diagram and join the dots. 

Results: The scoring can give you an overall score (0-50) and a diagram 
illustrating how open (pragmatic) or restrictive (explanatory) the study 
is likely to be. 

Interpretation: Figure 7 (see main text) demonstrates an explanatory study 
investigating whether the experimental intervention will work in ideal cir- 
cumstances (total score 0-1 5) and a more pragmatic study focusing mostly 
on whether, in routine practice, an intervention has a meaningful effect 
(total score >35). A total score between 16 and 35 suggest an interim 
where trail design balances pragmatic and explanatory domains. 
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. Eligibility criteria 

• Does the study population represent the people who will 
really be treated with the experimental intervention? 

• Think about restrictions to participation - such as a 
pre-determined number of participants or any exclusion 
criteria which make the trial less real-world. 

• Consider any selection bias toward high risk people or 
those without co-morbidity. 

• Trials open to the entire target population are more 
pragmatic, those which have exclusion criteria become 
progressively less so as exclusions become more rigid. 



Explanatory pragmatic 
0 12 3 
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2. Flexibility of experimental intervention 

• Does application of the intervention allow for variation 
and adaptability which exists in everyday life? 

• Think about rigidity of protocol application and strict, 
extensive or complex instructions - these make a study less 
pragmatic. 

• Less pragmatic still are trials which closely monitor 
application of intervention and make exclusions based on 
this. 

I* If creativity is allowed or encouraged in applying the 
intervention and/or it is tailored to fit the individual - this 
is more representative of real life and more pragmatic. 
Explanatory pragmatic 

0 1 2 3 4 5 



. Practitioner expertise 
- for experimental intervention 

• Can the experimental intervention be carried out by 
people ordinarily involved with the care of participants? 

• Less pragmatic interventions require higher levels of 
practitioner expertise - this can mean staff education or 
training (intensity is important) or bringing in others not 
normally involved in the care of participants. 

• Sometimes the intervention requires involvement of an 
external expert - particularly non-pragmatic. 
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4. Practitioner expertise 

- for comparison intervention 

• Can the comparison be carried out by people ordinarily 
involved with the care of participants? 

• Less pragmatic interventions require higher levels of 
practitioner expertise - this can mean staff education or 
training (intensity is important) or bringing in others not 
normally involved in the care of participants. 

• Sometimes the intervention requires involvement of 
external expert - particularly non-pragmatic 
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5. Flexibility of comparison 

• Is the intervention being compared with what is currently 
happening in everyday practice? The level of pragmatism is 
reduced by comparing the experimental intervention to 
other structured or regulated comparisons. 

• The ideal pragmatic comparison is unregulated treatment 
as usual. 

• The more a comparison diverges from this by introducing 
regulation, instruction or monitoring - the more 
explanatory the trial becomes. 
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7. Participant compliance 

• In real life compliance with treatment is highly variable, 
does this trial falsely eliminate non-compliance by 
monitoring or by excluding those who falter in 
compliance? 

• Pragmatic trials accept that non-compliance is a fact of life 
and therefore avoid monitoring, or trying to improve 
compliance. 

• Ideally pragmatic trials should avoid measuring compliance 
altogether since any information obtained could be used 
future trials to falsely improve compliance. 
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6. Outcomes 

• Are outcomes important to the health/needs of 
participants (pragmatic) or the needs/profits of the 
researcher (explanatory)? 

• How complex is the measurement of outcomes? 

• How much does measurement of outcome require input 
from participants? 

• More pragmatic trials will look at outcomes in the long 
term and no specific training will be required for 
measurement. 

Pragmatism should also avoid the need for central 
adjudication of outcome measurement. 
The outcome should be positive for the participant. 
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8. Practitioner adherence 

• Does this study regulate adherence to treatment to a 
degree which would not be replicated in real life? 

• Pragmatic trials should avoid monitoring practitioner 
adherence to the study protocol, explanatory trials would 
try to identify practitioners with poor compliance and 
possibly eliminate their data from the study. 

• Between these two extremes trials which try to improve 

■adherence are less pragmatic and those which allow for 
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9. Analysis of outcomes 

• Have they excluded any data? Exclusion of data often shifts 
a trial away from pragmatism because it has the effect of 
making the trial an ideal situation rather than a real one. 

• For optimum pragmatism the analysis should avoid looking 
at subgroups or excluding data because of practitioner 
adherence, participant compliance or other factors. 

• More data exclusion = less pragmatic. 
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10. Follow-up intensity 

• Intense follow-up is not a realistic scenario. The pragmatic 
approach to follow-up is covert and would occur during 
usual care preferably with as little disruption to the 
participant's normal routine as possible. 

• Follow up should be long term. 

• More explanatory follow-up measures may include: 
pre-specified appointments for the collection of data, short 
term follow up looking for a specific outcome, coercion or 
encouragement to attend follow-up and the collection of 
unusually large amounts of data not directly relevant to 
the participant. 

Explanatory pragmatic 
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Ensayos pragmaticos versus explicativos: 
Pragmascope, el instrumento que 
ayudar a medir las diferencias en los 
protocolos de ensayos controlados 
randomizados de salud mental 

En el continuo pragmatico-explicativo, un ensayo 
controlado randomizado (ECR) puede investigar por 
una parte si un tratamiento podna funcionar en cir- 
cunstancias ideales (explicativas) y por otra si podria 
funcionar en la practica diaria (pragmatico). Cuan 
explicativo o pragmatico sea un estudio, puede 
tener repercusiones para los clinicos, los politicos, los 
pacientes, los investigadores, los organismos de 
financiamiento y el publico. Hay una necesidad cre- 
ciente de que los estudios sean abiertos y pragmati- 
cos; sin embargo, tambien son necesarios los ensa- 
yos explicativos. PRECIS, el resumen del indicador del 
continuo pragmatico-explicativo que se habfa desa- 
rrollado con anterioridad, se adapto al instrumento 
Pragmascope para ayudar a los investigadores de 
salud mental en los disenos de ECR, tomando en 
consideracion el continuo pragmatico-explicativo. 
Se eligieron randomizadamente diez protocolos de 
ensayos de salud mental y se les asigno puntaje 
mediante el instrumento por tres evaluadores inde- 
pendientes. Sus resultados se compararon en cuanto 
a consistencia y se encontro que el instrumento fue 
confiable y practico. Este trabajo preliminar sugiere 
que es util la evaluation de diferentes aspectos de 
un ECR a nivel del protocolo y propone que el 
empleo del instrumento Pragmascope presentado 
aqufsen'a una forma practica de hacerlo. 



Etudes pragmatiques versus explicatives : 
I'outil Pragmascope de mesure pour mesurer 
les differences dans les protocoles d'etudes 
controlees randomisees sur la sante mentale 

Dans le continuum pragmatique-explicatif, une 
etude controlee randomisee (ECR) peut d'un cote 
analyser I'efficacite d'un traitement en circonstances 
ideales (explicatives) ou d'un autre cote son effica- 
cite dans la pratique quotidienne (pragmatique). 
Qu'une etude soit explicative ou pragmatique, elle 
peut avoir des implications pour les medecins, les 
responsables politiques, les patients, les chercheurs, 
les organismes de financement et le public. Les 
etudes doivent de plus en plus etre ouvertes et prag- 
matiques, mais les etudes explicatives sont aussi 
necessaires. L'indicateur de continuum pragmatique 
- explicatif PRECIS (Pragmatic-explanatory conti- 
nuum indicator summary) developpe anterieure- 
ment a ete adapte sous forme de I'outil PRAGMA- 
SCOPE pour aider les chercheurs en sante mentale a 
concevoir des ECR, en prenant en compte led it 
continuum. Dix protocoles d'etudes en sante men- 
tale ont ete choisis et cotes au hasard grace a I'ou- 
til, par trois evaluateurs independants. Leurs resul- 
tats ont ete compares pour evaluer leur coherence 
et I'outil s'est avere fiable et pratique. Ce travail pre- 
liminaire suggere que /'evaluation des differents 
domaines des ECR au niveau du protocole est utile 
et que /'utilisation du PRAGMASCOPE presente id 
pourrait etre un moyen pratique de le faire. 
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